Pravin Prakashrao Kulkarni¶
In [1]:
# Import Libraries
import warnings
warnings.filterwarnings('ignore')
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set(color_codes=True)
from sklearn import preprocessing
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from scipy.cluster.hierarchy import dendrogram, linkage, cophenet, fcluster
from scipy.spatial.distance import cdist, pdist
import plotly.express as px
Load Data¶
In [2]:
cardData = pd.read_excel('Credit Card Customer Data.xlsx')
cardData.head()
Out[2]:
Deliverable – 1¶
Univariate analysis¶
In [3]:
# Shape of data
cardData.shape
Out[3]:
In [4]:
rowCount = cardData.shape[0]
rowCount
Out[4]:
In [5]:
# Data Types
cardData.dtypes
Out[5]:
In [6]:
# Null check
cardData.isnull().sum()
Out[6]:
In [7]:
# number of unique values
cardData.nunique()
Out[7]:
In [8]:
# Unique values & counts i.e. value counts
for col in cardData.columns:
print('----------- Value Counts of ', col, ' -----------')
print(cardData.groupby(by=col)[col].value_counts())
print('-------------------------------------')
print()
In [9]:
# Statistical summary of numeric columns
cardData.describe().T
Out[9]:
In [10]:
def getQuartileCounts(col):
print('Quartile Analysis of ', col)
print()
Min = cardData[col].min()
Max = cardData[col].max()
Std = cardData[col].std()
Mean = cardData[col].mean()
Median = cardData[col].median()
Q1 = cardData[col].quantile(q=0.25)
Q2 = cardData[col].quantile(q=0.50)
Q3 = cardData[col].quantile(q=0.75)
IQR = Q3-Q1
Minimum = Q1 - (1.5 * IQR)
Maximum = Q3 + (1.5 * IQR)
LeftOutlierCount = cardData[cardData[col]<Minimum].shape[0]
Q1Count = cardData[(cardData[col]>=Minimum) & (cardData[col]<=Q1)].shape[0]
Q2Count = cardData[(cardData[col]>Q1) & (cardData[col]<=Q2)].shape[0]
Q3Count = cardData[(cardData[col]>Q2) & (cardData[col]<=Q3)].shape[0]
Q4Count = cardData[(cardData[col]>Q3) & (cardData[col]<=Maximum)].shape[0]
RightOutlierCount = cardData[cardData[col]>Maximum].shape[0]
print('Min ', Min, " Value count: ", cardData[cardData[col] == Min].shape[0])
print('Max ', Max, " Value count: ", cardData[cardData[col] == Max].shape[0])
print()
print('Standard Deviation ', Std)
print('Mean ', Mean)
print('Median ', Median)
print()
print('25th percentile Q1 ', Q1)
print('50th percentile Q2 ', Q2)
print('75th percentile Q3 ', Q3)
print('IQR ', IQR)
print('Minimum = Q1 - 1.5*IQR = ', Minimum)
print('Maximum = Q3 + 1.5*IQR = ', Maximum)
print()
print('Left outlier count i.e. < Minimum ', LeftOutlierCount)
print('>= Minimum and <= Q1 count ', Q1Count)
print('> Q1 and <= Q2 count ', Q2Count)
print('> Q2 and <= Q3 count ', Q3Count)
print('> Q3 and <= Maximum ', Q4Count)
print('Right outlier count i.e. > Maximum ', RightOutlierCount)
print('Total Outliers ', (LeftOutlierCount+RightOutlierCount), ' ', (LeftOutlierCount+RightOutlierCount)*100/rowCount, '%', ' of total records')
print()
if(Mean > Median):
print('Distribution is Right Skewed because Mean > Median')
elif(Mean < Median):
print('Distribution is Left Skewed because Mean < Median')
else:
print('Distribution is Symmetric because Mean = Median')
def plotUnivariate(col, pltType):
if (pltType == 'box-dist'):
fig, axes = plt.subplots(1, 2, figsize=[15,5])
fig.tight_layout(pad=5.0)
getQuartileCounts(col)
sns.boxplot(data=cardData, x=col, ax=axes[0]);
sns.distplot(cardData[col], ax=axes[1]);
In [11]:
# Plot for Avg_Credit_Limit
plotUnivariate('Avg_Credit_Limit', 'box-dist')
Looking at number of peaks in the distribution, 3 clusters can be formed based on Average Credit limit.¶
In [12]:
# Plot for Total_Credit_Cards
plotUnivariate('Total_Credit_Cards', 'box-dist')
Looking at number of peaks in the distribution, 5 clusters can be formed based on Total Credit Cards.¶
In [13]:
# Plot for Total_visits_bank
plotUnivariate('Total_visits_bank', 'box-dist')
Looking at number of peaks in the distribution, 3 clusters can be formed based on Total Bank Visits.¶
In [14]:
# Plot for Total_visits_online
plotUnivariate('Total_visits_online', 'box-dist')
Looking at number of peaks in the distribution, 4 clusters can be formed based on Total Online Visits.¶
In [15]:
# Plot for Total_calls_made
plotUnivariate('Total_calls_made', 'box-dist')
Looking at number of peaks in the distribution, 2 clusters can be formed based on Total Calls Made.¶
In [16]:
# Let's drop the Serial Number and Customer Key columns (because these are Identifier columns)
# and create new dataset for analysis.
cardDataTrimmed = cardData.drop(columns=['Sl_No', 'Customer Key'])
In [17]:
# Correlation between the columns
cardDataTrimmed.corr()
Out[17]:
In [18]:
# Pair plot
sns.pairplot(cardDataTrimmed, diag_kind='kde');
Observations¶
Weak correlations (less than ± 30%) are ignored here.¶
- Total Credit Cards and Average Credit Limit are positively correlated. Degree of Correlation 0.608860
- Total Credit Cards and Total Bank Visits are positively correlated. Degree of Correlation 0.315796
- Total Visits Online and Average Credit Limit are positively correlated. Degree of Correlation 0.551385
- Total Visits Online and Total Bank Visits are negatively correlated. Degree of Correlation -0.551861
- Total Calls Made and Average Credit Limit are negatively correlated. Degree of Correlation -0.414352
- Total Calls Made and Total Credit Cards are negatively correlated. Degree of Correlation -0.651251
- Total Calls Made and Total Bank Visits are negatively correlated. Degree of Correlation -0.506016
Create new feature based on various types of contact and visualize in 3D¶
In [19]:
# Let's create a new feature based on different types of contacts
# This feature will help in segmentation and visualization
cardDataTotalContactsCopy = cardDataTrimmed.copy()
cardDataTotalContactsCopy["Total_Contacts"] = (cardDataTotalContactsCopy["Total_visits_bank"]
+ cardDataTotalContactsCopy["Total_visits_online"]
+ cardDataTotalContactsCopy["Total_calls_made"])
cardDataTotalContactsCopy.head()
Out[19]:
In [20]:
# Scatter 3D Plot
px.scatter_3d(cardDataTotalContactsCopy, x='Avg_Credit_Limit', y='Total_Credit_Cards', z='Total_Contacts',
color='Total_Contacts')
Deliverable – 2¶
K-means Clustering¶
In [21]:
random = 7
In [22]:
# Data Preprocessing - Scaling
scalar = preprocessing.StandardScaler()
cardDataKMeans = pd.DataFrame(scalar.fit_transform(cardDataTrimmed), columns=cardDataTrimmed.columns)
cardDataKMeans.head()
Out[22]:
In [23]:
# K-means - Let's find the k using Average Distortions
distortions = []
cardDataDistortionsCopy = cardDataKMeans.copy()
kMeansClusters = range(1, 15)
for k in kMeansClusters:
model=KMeans(n_clusters=k, random_state=random)
model.fit(cardDataDistortionsCopy)
prediction=model.predict(cardDataDistortionsCopy)
distances = cdist(cardDataDistortionsCopy, model.cluster_centers_, 'euclidean')
minDistance = np.min(distances, axis=1)
averageMinDistance = sum(minDistance)/rowCount
distortions.append(averageMinDistance)
print('Average Distortions')
print()
print(distortions)
fig = plt.figure(figsize=[20,5])
plt.plot(kMeansClusters, distortions, 'bo-', )
plt.xlabel('k')
plt.ylabel('Average distortion')
plt.title('Selecting k with the Elbow Method - Average Distortions');
From the elbow graphs above, it can be seen that either 4 or 5 clusters can be formed.¶
In [24]:
# K-means - Let's find the k using Silhouette Score
silhouette = []
cardDataSilhouetteCopy = cardDataKMeans.copy()
kMeansClusters = range(2, 15)
for k in kMeansClusters:
model=KMeans(n_clusters=k, random_state=random)
model.fit(cardDataSilhouetteCopy)
prediction=model.predict(cardDataSilhouetteCopy)
silhouette.append(silhouette_score(cardDataSilhouetteCopy, prediction))
print('Silhouette Score')
print()
print(silhouette)
plt.plot(kMeansClusters, silhouette, 'bo-')
plt.xlabel('k')
plt.ylabel('Silhouette Score')
plt.title('Selecting k with the Elbow Method - Silhouette Score');
From the elbow graphs above, it can be seen that 5 or 6 clusters can be formed.¶
In [25]:
cardDataFinalModelCopy = cardDataKMeans.copy()
model=KMeans(n_clusters=5, random_state=random)
model.fit(cardDataFinalModelCopy)
labels=model.predict(cardDataFinalModelCopy)
print("Silhouette Score for Final KMeans model:", silhouette_score(cardDataFinalModelCopy, labels))
cardDataFinalModelCopy['Group'] = labels
cardDataFinalModelCopy.head()
Out[25]:
In [26]:
# Scatter 3D Plot
px.scatter_3d(cardDataFinalModelCopy, x='Avg_Credit_Limit', y='Total_Credit_Cards', z='Group', color='Group')
In [27]:
# Box Plot
cardDataFinalModelCopy.boxplot(by='Group', figsize=(15,15));
In [ ]:
In [ ]:
Deliverable – 3¶
Hierarchical Clustering¶
In [28]:
# Data Preprocessing - Scaling
scalar = preprocessing.StandardScaler()
cardDataHierarchical = pd.DataFrame(scalar.fit_transform(cardDataTrimmed), columns=cardDataTrimmed.columns)
cardDataHierarchical.head()
Out[28]:
In [29]:
#linkageMethods=['single','complete','average','ward','median']
hierarchicalResults = pd.DataFrame(columns=['Linkage Method', 'Cophenetic Correlation', 'Dendogram Distance For 5 Clusters',
'Silhouette Score'])
In [30]:
def plotHierarchicalClustering(mtd, dataSet):
linkageMatrix = linkage(dataSet, method=mtd, metric='euclidean')
cophenetic_correlation, cophenetic_distances = cophenet(linkageMatrix, pdist(dataSet))
print("Linkage Method:", mtd)
print("Cophenetic Correlation:", cophenetic_correlation)
plt.figure(figsize=(25, 10))
plt.title('Hierarchical Clustering Dendrogram - showing Top 15 clusters for Linkage Method "'+mtd+'"',
fontdict={'fontsize': 25})
dendrogram(linkageMatrix, truncate_mode='lastp', p=15)
dendogramDistance = 2
clusters = fcluster(linkageMatrix, dendogramDistance, criterion='distance')
if(len(set(clusters)) < 5):
while(len(set(clusters)) != 5):
dendogramDistance -= 0.01
clusters = fcluster(linkageMatrix, dendogramDistance, criterion='distance')
elif(len(set(clusters)) > 5):
while(len(set(clusters)) != 5):
dendogramDistance += 0.01
clusters = fcluster(linkageMatrix, dendogramDistance, criterion='distance')
print("Dendogram Distance where ", len(set(clusters)), " clusters found is:", dendogramDistance)
silhouetteScore = silhouette_score(dataSet, clusters)
print("Silhouette Score:", silhouetteScore)
dataSet['Group'] = clusters
# Box Plot
dataSet.boxplot(by='Group', figsize=(15,15));
resultFrame = pd.Series([mtd, cophenetic_correlation, dendogramDistance, silhouetteScore], index=hierarchicalResults.columns)
return resultFrame
In [31]:
result = plotHierarchicalClustering('single', cardDataHierarchical.copy())
hierarchicalResults = hierarchicalResults.append(result, ignore_index=True)
In [32]:
result = plotHierarchicalClustering('complete', cardDataHierarchical.copy())
hierarchicalResults = hierarchicalResults.append(result, ignore_index=True)
In [33]:
result = plotHierarchicalClustering('average', cardDataHierarchical.copy())
hierarchicalResults = hierarchicalResults.append(result, ignore_index=True)
In [34]:
result = plotHierarchicalClustering('ward', cardDataHierarchical.copy())
hierarchicalResults = hierarchicalResults.append(result, ignore_index=True)
In [35]:
result = plotHierarchicalClustering('median', cardDataHierarchical.copy())
hierarchicalResults = hierarchicalResults.append(result, ignore_index=True)
In [36]:
hierarchicalResults.sort_values(by='Cophenetic Correlation', ascending=False)
Out[36]:
From above, Cophenetic Correlation & Silhouette Score for Average Linkage is maximum, so that is final Linkage method for Hierarchical Clustering¶
In [ ]:
In [ ]:
Deliverable – 4¶
Silhouette score for final models¶
In [37]:
# KMeans - from model finalized above
model=KMeans(n_clusters=5, random_state=random)
model.fit(cardDataKMeans)
labels=model.predict(cardDataKMeans)
print("Silhouette Score for Final KMeans model:", silhouette_score(cardDataKMeans, labels))
In [38]:
# Hierarchical Clustering - from linkage method, dendogramic distance finalized above
linkageMatrix = linkage(cardDataHierarchical, method='average', metric='euclidean')
clusters = fcluster(linkageMatrix, 2.51, criterion='distance')
print("Silhouette Score for Final Hierarchical clustering:", silhouette_score(cardDataHierarchical, clusters))
Deliverable – 5¶
In [39]:
# Assign groups on original trimmed data from final model labels in above steps.
cardDataCopyKMeans = cardDataTrimmed.copy()
cardDataCopyKMeans['Group'] = labels
cardDataCopyHierarchical = cardDataTrimmed.copy()
cardDataCopyHierarchical['Group'] = clusters
For K-means clusters - Find Statistical summery for each group formed and see the distribution for each column across the groups.¶
In [40]:
cardDataCopyKMeans.groupby(by='Group').describe().T
Out[40]:
In [41]:
cardDataCopyKMeans.groupby(by='Group')['Avg_Credit_Limit'].plot(kind='kde');
In [42]:
cardDataCopyKMeans.groupby(by='Group')['Total_Credit_Cards'].plot(kind='kde');
In [43]:
cardDataCopyKMeans.groupby(by='Group')['Total_visits_bank'].plot(kind='kde');
In [44]:
cardDataCopyKMeans.groupby(by='Group')['Total_visits_online'].plot(kind='kde');
In [45]:
cardDataCopyKMeans.groupby(by='Group')['Total_calls_made'].plot(kind='kde');
For Hierarchical clusters - Find Statistical summery for each group formed and see the distribution for each column across the groups.¶
In [46]:
cardDataCopyHierarchical.groupby(by='Group').describe().T
Out[46]:
In [47]:
# Remove two records from group 3 and 5 to avoid error during plot as it requires more than one records.
df_For_Graph = cardDataCopyHierarchical[(cardDataCopyHierarchical["Group"] != 3) & (cardDataCopyHierarchical["Group"] != 5)]
In [48]:
df_For_Graph.groupby(by='Group')['Avg_Credit_Limit'].plot(kind='kde');
In [49]:
df_For_Graph.groupby(by='Group')['Total_Credit_Cards'].plot(kind='kde');
In [50]:
df_For_Graph.groupby(by='Group')['Total_visits_bank'].plot(kind='kde');
In [51]:
df_For_Graph.groupby(by='Group')['Total_visits_online'].plot(kind='kde');
In [52]:
df_For_Graph.groupby(by='Group')['Total_calls_made'].plot(kind='kde');
Observations¶
- In K-Means, Group 0 and 3 have similar number of items and group 1 and 2 have similar. However, in Hierarchical, group counts are not similar. Thus Hierarchical clustering distinguishes the groups correctly.
- In Hierarchical, group 3 and 5 have only one record, thus able to find outliers.
- Hierarchical clustering tries to minimize the mixture of Gausians across the clusters formed.
- Though Hierarchical clustering gives more accurate results, we have to keep the computational requirements in mind while making decision whether to use Hierarchical clustering or not.
Deliverable – 5¶
Answers to Key questions¶
As the Hierarchical clustering proved to be best in this case, below answers are based on the Hierarchical clustering and excluding the outliers. (i.e. group 3 and 5)¶
-
How many different segments of customers are there?
There are 3 different segments of customers.
-
How are these segments different from each other?
-
Customers with higher credit limit (group 1)
- Have 5 to 10 credit cards.
- Visits bank very less or never.
- Prefer online visits.
- Calls bank very less or never.
-
Customers with moderate credit limit (group 4)
- Have 4 to 7 credit cards.
- Visits bank at least once.
- Visit online very less or never.
- Calls bank very less or never.
-
Customers with lower credit limit (group 2)
- Have 1 to 4 credit cards.
- Visits bank very less or never.
- Visit online at least once.
- Calls bank 5 times or more.
-
Customers with higher credit limit (group 1)
-
What are your recommendations to the bank on how to better market to and service these customers?
- As customers having higher credit limit do visit online, try to contact through e-advertising, emails, voice/text messages etc.
- As customers having moderate credit limit visit bank at least once, talk during their visits to bank, provide information about the schemes answer their queries and try to convert the opportunities.
- As customers having lower credit limit calls more often and also visits online at least once, explaining about bank schemes during the call or playing pre-recorded messages during their wait time while connecting with an executive of customer care will help to maximize the conversion of opportunities. And also try to contact through e-advertising, emails, voice/text messages etc.
In [ ]: